Multimodal Vision-Language Models (VLMs) enable powerful applications from their fused understanding of images and language, but many perform poorly on UI tasks due to the lack of UI training data. In this paper, we adapt a recipe for generating paired text-image training data for VLMs to the UI domain by combining existing pixel-based methods with a Large Language Model (LLM). Unlike prior art, our method requires no human-provided annotations, and it can be applied to any dataset of UI screenshots. We generate a dataset of 335K conversational examples paired with UIs that cover Q&A, UI descriptions, and planning, and use it to fine-tune a conversational VLM for UI tasks. To assess the performance of our model, we benchmark it on UI element detection tasks, evaluate response quality, and showcase its applicability to multi-step UI navigation and planning.

While the Genius of Things event in Boston is over, I'm still pondering. There are so many interesting ways that our customers are using Watson IoT to change the way we live and work. In addition to the great speakers, there were also some very impressive technology demonstrations that our partners and colleagues brought to the event. We did, when we stopped by to chat with the Persistent team. Persistent has been a leader in developing humanoid robots with Watson IoT.